Web services are considered as foundational components to service oriented architecture (SOA). As more and more enterprises are deploying more and more web services, and services ecosystems grow to connect more and more parties, the importance attributed to the quality of web services is becoming incontrovertible. It is no surprise that web service testing is one of the first things that is thought of when considering SOA testing. Currently there are quite a lot of tools with web service testing capability in the market or open source community: to list a few, Parasoft SOATest, Mindreef SOAPScope, CrossCheck Networks SOAPSonar, SOAPUI.
Generally these tools provide a graphical editor that allows users to import web services definition language (WSDL) definition, edit test cases and run the test cases. A test case has an input message that will be used as a parameter to invoke the service under test, and an output message that is expected from the service. The actual output message is compared to the expected one. Based on the comparison, a judgment is made on whether or not the service function is correct.
For most of these tools, there is no automatic test message/data generation capability. A test message refers to a message that tells what parameter values to use in testing a web service operation. Users need to input the contents of the test messages manually from the user interface. This will lead to time-consuming test case design process. Consider a simple “add” function that takes two parameters x and y. To adequately test this function, many variations on the parameters need to be tried: x=1, y=1; x=−1, y=1; x=0, y=1; x=−1, y=−1; x=a maximum value for the integer int type, y=1; x=non-integer value, y=1; etc. This is only a simple case that involves only two parameters. Usually, a real web service operation uses large messages with complex structures. Tens or hundreds of fields in a single message is common. Thus, it would take a lot of effort to build a test case design manually.
Other existing tools generate test cases automatically from the WSDL definition. SOAPSonar utilizes XSD-Mutation, a patent-pending automation technique, to generate a set of test cases, both positive and negative. The test mutations may occur at the data type, data value, message structure, or protocol binding level. An example mutation is a buffer overflow type boundary conditions for an unbounded string type field. The test cases generated by SOAPSonar are mostly random test cases and may be ineffective in detecting bugs as it only refers to the data structure and type information defined in the WSDL. In reality, a WSDL definition usually does not have much information about the “real” type and value space of a field. For example, the parameters of an add operation are all with a “string” type, which gives no clue for a test generation algorithm to search in the numeric type value space. It can only generate random strings based on this type definition. These test cases may be ineffective in detecting bugs as most of the inputs are filtered in the service implementation before real business logic begins.
Rational Tester for SOA Quality (RTSQ)™ from International Business Machines Corporation (IBM) can generate a random test message from a WSDL definition, e.g., ‘abc’ for each string type field. The primary usage of such test cases is for performance testing, where message values are not important. This tool cannot be applied to the functional testing of web services.
In the object-oriented world, there are tools that can automatically generate test cases from method declarations. For example, Parasoft Jtest can do Java class unit test generation—test each method with various boundary values of each input parameter (boundary values for string type: null, empty, special characters, etc.), observe whether runtime exception occurs. It is not applicable to web service test generation because 1) there is the same type problem of WSDL definition as mentioned earlier; 2) further schema-compliance is needed in Web service test data, i.e., the input message must comply to its schema definition. WSDL and Extensible Markup Language (XML) schema definition (XSD) standards allow value space restriction on specific types, which can be used for validation of the XML. For example, for integer (int) type, max and min value can be specified; for string type, max and min length can be specified. Such information cannot be defined in Java method declaration, thus not considered in any test generation method.